This error message is very rare and is caused by insufficient memory available on Windows.
Quit the running program and record or make a smaller recording area.
Or it may be caused by setting the recording-scaling-video scaling resolution too high (for example, 8k resolution) in Ocam's settings.